Search for: All records

Sign language is a priceless means of communication for deaf and hard-of-hearing people to fully enable them to participate in society and interact with others. This study introduces a novel universal sign language system that uses the Gesture-script to generate a detailed description of gestures in videos, which involve continuous movement of hands, arms, heads, and body language. Subsequently, we input this description into a Large Language Model (LLM) to interpret sign language. We deployed a few-shot prompting technique for LLM, enabling it to precisely transfer the sign videos into corresponding sentences in natural language. Furthermore, the Few-shot prompting technique enables our system to interpret multiple types of sign language without pre-training or fine-tuning. 

This paper presents the MAE model that uses a Masked AutoEncoder (MAE) to enhance the observations from commercial passive Radio-Frequency Identification (RFID) devices. It is crucial to address the common issue of RFID readers failing to collect observations from all their hop channels and antennas due to environmental effects and device limitations. The proposed method examines the inner rationale among observations from various channels and antennas to reconstruct the missing observations, which can significantly improve the performance of downstream applications. The experiment results show that when we collect more than 70% observation in all antennas at all channels, our MAE model can restore 90% of the missing phase with an error of less than 0.1 radians, which is even less than the error caused by thermal noise in an RFID system. Our MAE model's accuracy in restoring missing data provides a promising future to improve the performance of various RFID applications like localization and motion tracking by providing more complete observations. 

This project introduces a framework to enable robots to recognize human hand signals, a reliable and feasible device-free means of communication in many noisy environments such as construction sites and airport ramps, to facilitate efficient human-robot collaboration. Various hand signal systems are accepted in many small groups for specific purposes, such as Marshalling on airport ramps and construction site crane operations. Robots must be robust to unpredictable conditions, including various backgrounds and human appearances, an extreme challenge imposed by open environments. To address these challenges, we propose Instant Hand Signal Recognition (IHSR), a learning-based framework with world knowledge of human gestures embedded, for robots to learn novel hand signals in a few samples. It also offers robust zero-shot generalization to recognize learned signals in novel scenarios. Extensive experiments show that our IHSR can learn a novel hand signal in only 50 samples, which is 30+ times more efficient than the state-of-the-art method. It also demonstrates a robust zero-shot generalization for deploying a learned model in unseen environments to recognize hand signals from unseen human users.